Observability for AI Workloads: A Primer

Observability and AI

Artificial Intelligence (AI) s a set of techniques, methods, and strategies that lets computers perform complex tasks that require skills usually associated with human beings, such as learning, reasoning, problem-solving, and decision-making.

A subset of AI (focused on generating content, regardless of the format) is Generative AI (GenAI). GenAI solutions have been deployed in organizations with the hopes of accelerating delivery and driving innovation, but not without challenges. One of the key aspects neglected in GenAI  is proper Governance. Effective governance relies on observability, strategy, and leadership.  Observability is a particularly interesting challenge for GenAI because these workloads are non-deterministic by nature and therefore require a high degree of observability maturity.

Why Observability?

You can understand and predict a system’s behavior through observability dashboards. You measure this behavior using signals and classify these signals into logs, traces, and metrics. GenAI solutions are non-deterministic. This means that constant monitoring is crucial to ensure costs stay within ranges. System behavior must align with organizational goals.

• Observability is a holistic view of system behavior. It monitors signals and provides insights.

• Monitoring collects, stores, and represents a system’s signals.

• Signals are variables you use to comprehend system health. They classify into logs, traces, and metrics.

• Logs are registries of events inside the system.

• Traces register operations spanning distributed systems. They provide context to understand system behavior.

• Metrics measure the system over time. They give insights into processing amount and type.

• Instrumentation prepares applications to provide observability signals. You then collect, process, and store the metrics.

• Dashboards represent the relevant signals defining system health.

• Topology is a graphical representation of component interaction.

• Time travel monitors and debugs a system at any historical point.

With system observability, you gain control and can make data-driven decisions. The ultimate goal is gaining insights to optimize and correct system behavior.

Generative AI

Prior to reading this blog, you probably already understand GenAI’s value to your business. Let’s review some concepts needed to understand GenAI observability.

• A Large Language Model (LLM), or model, processes prompts and performs generations.

• A prompt is an instruction that triggers the generation process.

• Generation happens when a model composes a solution. It uses internal or external knowledge.

• Multi-modal models can handle more than one format. Examples include text, images, and video.

• Number of parameters is like the synapses in a brain. Models with more parameters perform better. They are also more resource intensive.

• Reasoning models decompose problems into smaller ones. They achieve better results but process more tokens.

• Tokens are how LLMs encode information. This happens during prompt input, reasoning, or generation.

• Vector databases handle unstructured data as embeddings. This suits GenAI and RAG knowledge bases.

• Embeddings are representations of information in a vector database.

• RAG uses knowledge base information to enrich LLM generation context. It typically uses a vector database for storage.

• GenAI Stack is the set of applications that compose your GenAI product.

Why know the differences? Different models will have different costs and drawbacks. Reasoning models provide thoughtful responses but increase latency and cost. Ironically, complex tasks may cost less with expensive reasoning models. This avoids multiple, less effective user interactions. RAG-powered systems can also impact cost because they add knowledge management and data storage factors. Operating the system in a way where we can find a balance between correctness, performance, and cost is something only achievable in systems with a high degree of observability maturity.

Let’s explore what to observe, from physical to abstract layers.

First, acknowledge that GenAI requires specialized hardware. GPU monitoring is essential.  Through observability you can analyze metrics like temperature, power consumption, and usage. This ensures your infrastructure can handle the workloads.

GenAI applications are primarily software components. You can apply standard monitoring strategies here. Variables will include uptime, hardware utilization, and networking. They also include Kubernetes cluster allocation and other generic metrics.

Finally, consider GenAI-specific metrics. These are domain-specific metrics. Token consumption, for example, helps you understand consumption of reasoning, input, and output tokens, which help understand the full model utilization profile and estimate costs. AI observability also  includes vector database metrics. This evaluates insertion/retrieval latencies and data collection growth. We can compute aggregate results from base metrics.

With the base metrics, we can start digging into computing aggregate results, with dashboards demonstrating the relation between different aspects of the GenAI solution, offering all the building blocks for total observability.

Open Source and Open Standards

Open Source requires source code availability for study. License agreements formalize this philosophy. Its greater impact comes from creating collaborative communities. A unified vision and knowledge sharing drive innovation. The open-source movement drives significant software advances. These advances extend beyond the software itself. Organizations that handle open-source software emerged from this momentum.

Open Standards are technical guidelines. They ensure autonomy for adopting organizations. Using open standards helps companies avoid vendor lock-in. They also make interoperability a reality. Several excellent open-source tools power modern observability stacks.

• OpenTelemetry (OTEL) is a collection of tools. It enables you to generate, collect, and export signals. It is compatible with many standards.

  • OTLP describes the transport and delivery of telemetry data.

  • The OpenTelemetry Collector processes and exports telemetry data. It is vendor-agnostic.

  • The OpenTelemetry Operator provides custom Kubernetes resource definitions. This helps you manage collector instances and auto-instrumentation.

• Prometheus is an open-source monitoring toolkit. It handles time-series metrics from various sources.

Using an AI Observability stack powered by OpenTelemetry future-proofs your system. No vendor lock-in; no single player dictates the rules.

SUSE adheres to this philosophy. OpenTelemetry provides the strong foundation for our observability offerings.

Putting Theory into Practice with SUSE

Ready to advance your AI Observability? SUSE can help you implement robust observability. It uses open standards and open-source software for GenAI workloads.

With SUSE AI, you get out-of-the-box GenAI observability. Minimal configuration is required. Leverage features like time travel and topological representations to help you debug even the most complex environments.

With enhanced observability you can gain insights about all the key aspects of your GenAI Stack, fully integrated with the observability employed for monitoring your other workloads, with the same user experience.

Conclusion

GenAI observability is ultimately about building trust. Monitoring behavior and managing costs are crucial. Ensuring safety and fairness will define success. Organizations can deploy GenAI with confidence. They do this by embracing open standards and observability.