Easily Compare LLMs Using OpenAI and Google Sheets

This workflow allows you to easily evaluate and compare the outputs of two language models (LLMs) before choosing one for production.

In the chat interface, both model outputs are shown side by side. Their responses are also logged into a Google Sheet, where they can be evaluated manually or automatically using a more advanced model.

Use Case

You're developing an AI agent, and since LLMs are non-deterministic, you want to determine which one performs best for your specific use case. This template is designed to help you compare them effectively.

How It Works

• The user sends a message to the chat interface.
• The input is duplicated and sent to two different LLMs.
• Each model processes the same prompt independently, using its own memory context.
• Their answers, along with the user input and previous context, are logged to Google Sheets.
• You can review, compare, and evaluate the model outputs manually (or automate it later).
• In the chat, both responses are also shown one after the other for direct comparison.

How To Use It

• Copy this Google Sheets template (File > Make a Copy).
• Set up your System Prompt and Tools in the AI Agent node to suit your use case.
• Start chatting! Each message will trigger both models and log their responses to the spreadsheet.

Note: This version is set up for two models. If you want to compare more, you’ll need to extend the workflow logic and update the sheet.

About Models

You can use OpenRouter or Vertex AI to test models across providers.
If you're using a node for a specific provider, like OpenAI, you can compare different models from that provider (e.g., gpt-4.1 vs gpt-4.1-mini).

Evaluation in Google Sheets

This is ideal for teams, allowing non-technical stakeholders (not just data scientists) to evaluate responses based on real-world needs.

Advanced users can automate this evaluation using a more capable model (like o3 from OpenAI), but note that this will increase token usage and cost.

Token Considerations

Since each input is processed by two different models, the workflow will consume more tokens overall.
Keep an eye on usage, especially if working with longer prompts or running multiple evaluations, as this can impact cost.

Define Models to Compare

This node defines the array of model IDs to be compared.

In this template, we compare two models using the OpenRouter API. You can modify the list by specifying the full model IDs you want to test.

Example: ["openai/gpt-4.1", "mistralai/mistral-large"]

If you're using a different LLM provider (like OpenAI directly, or Google Vertex AI), make sure to update the model IDs according to that provider's naming conventions.

Note: This template is built for two models. For more, you’ll need to adjust the workflow logic and the Google Sheet structure.

Set model, sessionId, chatInput, sessionIdBase

This node prepares the variables used during the loop that queries each model.

• model: The ID of the model being used in the current iteration.
• sessionId: A unique session key combining the original session ID and model name. This ensures memory isolation per model.
• chatInput: The user’s input message.
• sessionIdBase: The original session ID without any model-specific suffix. Used in Sheets to group evaluations from the same session.

AI Agent

This AI Agent is connected to OpenRouter Models. The model is selected dynamically from the variable {{$json.model}}, defined earlier.

Memory is isolated per model using the {{$('Set model, sessionId, chatInput, sessionIdBase').item.json.sessionId}} key.

⚠️ This agent currently has no system prompt or tools configured. If you want to test specific tasks, you must define them yourself to reflect realistic use cases.

Chat Memory Manager

This node handles retrieval of prior context for the chat session. It helps with qualitative evaluation by storing context that’s injected into the Google Sheet.

It shares memory with the AI Agent via the “Simple Memory” node.

You can switch to Redis or Postgres memory backends if needed.

Prepare Data for Chat and Google Sheets

This node sets the following fields:

• output: The model's response, formatted for chat display with visual separation to make comparison easier.
• chatInput: The user input that will be recorded in Google Sheets.
• model_answer: The actual answer from the model being evaluated.
• model: The name or ID of the model providing the answer, used for identifying performance.
• context: A history of the prior conversation (excluding the latest input). If it's the user's first message, a placeholder is used.
• sessionId: A unique session identifier combining model name and session, ensuring separate context windows for each model.
• sessionIdBase: The original user session ID (without model suffix), useful for grouping responses from different models in Sheets.

Add Model Results to Google Sheet

This Google Sheets step records both model responses along for evaluation.

⚠️ Depending on the length of model responses, you may need to adjust row height or column width.

The template includes basic evaluation fields (model_1_eval, model_2_eval) with a dropdown like:
"Good", "Correct", "Bad", but feel free to customize with more granular rating criteria.

Easily Compare LLMs Using OpenAI and Google Sheets

Purpose and structure

This folder contains the Easily Compare LLMs Using OpenAI and Google Sheets workflow. Its executable nodes use agent, aggregate, chatTrigger, googleSheets, lmChatOpenRouter, memoryBufferWindow, memoryManager, set, splitInBatches, splitOut, summarize. The table and connection map below are derived from the JSON, not an inferred guarantee of behavior.

Setup

Import inactive. Configure your own credentials and replace REPLACE_WITH_* values. Review external destinations and Code nodes before running.

Entry points

When chat message received

Verification

Sanitized source candidate; not runtime-tested. This overview is generated from node metadata. Read the folder README for the connection map, credential types, and license.

Attribution

https://github.com/Zie619/n8n-workflows/blob/94007c1445d9258a7da116646b79473e7c7c3282/workflows/Splitout/1790_Splitout_Summarize_Automation_Triggered.json MIT; preserve source notices.