AI/ML for Embodied Systems at the Edge: Generative Models, LLMs and Beyond
Andreas ANDREOU, Professor, Johns Hopkins University

Problem:

Operation and decision making at the EDGE for embodied applications such as autonomous robotics requires real-time local processing with extreme energy efficiency and low latency. Hardware must perform insightful information extraction from sensing signals to symbols, and distill knowledge into models necessary for reasoning and inference to generate action signals where computations are done by COTS or specialized hardware at the edge. A canonical model of processing in embodied systems is shown in the diagram on the right.

The Role of Generative AI in Robust Reasoning:

Generative AI such as Large Language Models (LLMs) can play a key role for real-time learning and Contextual Modeling so that subsequently the machine can perform robust reasoning and produce desired behavior (actions). Despite the recent explosion of advances in Large Language Models for text (ChatGPT) and images (Dall-E, Midjourney) generative AI, the computational structures “under the hood” can have a broad impact. For example, graphical models such as the Diffusion Models in Dalle-E or Deep Belief Networks (DBNs), are of generative nature consisting of multiple layers of nodes connected as Markov random fields where sampling plays a central role. The latter are computationally intensive necessitating micro-architectural components available on custom hardware such as SpiNNaker SOC Arm M4 chip multiprocessor2 or FPGAs. Action recognition without a camera at the edge4, necessitates the recognition of action using low dimensional data (time series of micro-Doppler acoustic signatures) but it is trained on signatures from high dimensional signals that can be generated from Kinect 3D cloud data or physical model (Knowledge). Continuous life-long learning to keep the Knowledge up to date necessitates generative AI and sampling in Conditional Restricted Boltzmann Machines (RCBMs) or Conditional Deep Belief Networks (CDBNs). An example is shown on the side where a micro-Doppler time series of an action (top) is used to seed a CDBN and the CDBN “hallucinates” the time series of response (bottom).

Generative AI for next Generation AI machines:

The natural language prompting of ChatGPT can also be employed to produce synthesizable Verilog for chip design. We have employed ChatGPT4 for natural language driven hardware design. The AI-generated design, a synthesizable and functional verilog description for the entirety of a programmable Spiking Neuron Array including an SPI interface, for neurocomputing at the edge. The latter was verified in simulation using handcrafted testbenches and is currently fabricated in Skywater 130nm CMOS technology through Tiny Tapeout 5 using an open-source EDA flow.