By Tim Leers
The term "LLMOps" (Large Language Model Operations) is debated - while it does encapsulate the operational challenges of deploying and managing large language models, it's the powerful, generative, and interactive nature of contemporary LLMs that present distinct challenges and opportunities. Moreover, the "large" aspect of LLMs that poses significant infrastructure challenges may be temporary as the open-source world continues to shrink model sizes. As such, the focus of "LLMOps" might shrink to subsets that more accurately describe the operations challenges:
GenerativeOps: The generative aspect of large language models, which refers to their ability to create new, coherent, and contextually relevant outputs. GenerativeOps emphasises the operational challenges and considerations associated with managing these generative capabilities, such as controlling the model's output, ensuring the quality and relevance of generated content, and monitoring for potential misuse.
Language Model Ops: This term captures the idea that the operations are specifically for language models. However, this term might be too broad, as it could also apply to simpler, non-generative language models that don't require the same level of operational complexity as large and/or generative ones.
AgentOps: This term focuses on the interactive nature of large language models, framing them as "agents" that can interact with users and the environment. It highlights the need for operations that support and manage these interactions, ensuring that the model behaves appropriately and provides value in its interactions with users and the environment. More broadly speaking, agents may become the next frontier of AI, where we let them take independent decisions and actions to accomplish tasks; this is where many of the concerns about existential AI risk originate, as without appropriate guardrails, such agents can go off the rails and take unintended paths to accomplishing task objectives. A significant challenge in AgentOps will therefore be managing the environment in which an agent operates, the ease with which it can access and manipulate resources, as well as how different agents can interact with each other, especially in scenarios where agents may have incompatible objectives.
While LLMOps is a fitting term given the operational aspect of dealing with these large-scale models, the focus of LLMOps may shift towards more narrow "Ops" challenges in the very near future, making the current term obsolete. Indeed, Agents and the associated engineering and operations challenges under "AgentOps" are set to become the next frontier for AI/ML solutions due to their ability to create unprecedented value at scale.