Aug 2, 2023

Langfuse Update — July 2023

Building out the core platform and preparing infrastructure for advanced analytics: Async SDKs, automated token counts, nested Trace UI, langchain integration, public GET API

Marc Klingen

Hi all, a lot has happened since our launch two weeks ago and I’m excited to share our July update:

• Faster integration
  • Langchain integration: Integrate with Langfuse in minutes when using Langchain
  • SDKs for Python and JS/TS: Now fully async, typed, and with improved DX for nested traces
• New nested Trace UI
• More metrics
  • Automated token counts for OpenAI and Anthropic models
  • Human-in-the-loop evaluation as an additional source of scores
• Analytics
  • User reporting on token usage, traces and feedback
  • Analytics (alpha) continuously rolling out to more users
• Other
  • Public GET API for custom integrations
  • Improved docs with new content and Q&A chatbot

The details 👇

🦜🔗 Langchain Integration

For teams building their LLM app with Langchain, adopting Langfuse just got that much easier. We added a CallbackHandler to the Langfuse Python SDK that automatically traces your complex Langchain chains and agents. Simply pass it as a callback.

pip install langfuse

# Initialize Langfuse handler
from langfuse.callback import CallbackHandler
langfuse_handler = CallbackHandler(
    secret_key="sk-lf-...",
    public_key="pk-lf-...",
    host="https://cloud.langfuse.com", # 🇪🇺 EU region
  # host="https://us.cloud.langfuse.com", # 🇺🇸 US region
)
 
# Your Langchain code
 
# Add Langfuse handler as callback (classic and LCEL)
chain.invoke({"input": "<user_input>"}, config={"callbacks": [langfuse_handler]})

chain.run(input="<user_input>", callbacks=[langfuse_handler]) # Legacy
conversation.predict(input="<user_input>", callbacks=[langfuse_handler])

pip install langfuse

LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_HOST="https://cloud.langfuse.com" # 🇪🇺 EU region
# LANGFUSE_HOST="https://us.cloud.langfuse.com" # 🇺🇸 US region

# Initialize Langfuse handler
from langfuse.callback import CallbackHandler
langfuse_handler = CallbackHandler()
 
# Your Langchain code
 
# Add Langfuse handler as callback (classic and LCEL)
chain.invoke({"input": "<user_input>"}, config={"callbacks": [langfuse_handler]})

chain.run(input="<user_input>", callbacks=[langfuse_handler]) # Legacy
conversation.predict(input="<user_input>", callbacks=[langfuse_handler])

npm i langfuse-langchain

import { CallbackHandler } from "langfuse-langchain";
// Deno: import CallbackHandler from "https://esm.sh/langfuse-langchain";
 
const langfuseHandler = new CallbackHandler({
  secretKey: "sk-lf-...",
  publicKey: "pk-lf-...",
  baseUrl: "https://cloud.langfuse.com", // 🇪🇺 EU region
  // baseUrl: "https://us.cloud.langfuse.com", // 🇺🇸 US region
});
 
// Your Langchain code
 
// Add Langfuse handler as callback to `run` or `invoke`
await chain.invoke({ input: "<user_input>" }, { callbacks: [langfuseHandler] });

npm i langfuse-langchain

LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_BASEURL="https://cloud.langfuse.com" # 🇪🇺 EU region
# LANGFUSE_BASEURL="https://us.cloud.langfuse.com" # 🇺🇸 US region

import { CallbackHandler } from "langfuse-langchain";
// Deno: import CallbackHandler from "https://esm.sh/langfuse-langchain";
 
const langfuseHandler = new CallbackHandler();
 
// Your Langchain code
 
// Add Langfuse handler as callback to `run` or `invoke`
await chain.invoke({ input: "<user_input>" }, { callbacks: [langfuseHandler] });

→ Integration docs

⚡️ Improved SDKs for Python and JS/TS

Most Langfuse users integrate using our Python or JS/TS SDKs. We made improvements to both DX and performance:

1. Fully async, avoid adding latency or blocking your app
2. Typed (Pydantic/Typescript)
3. Improved DX for nesting traces; no need to manually manage IDs
4. Support for more runtimes: JS/TS SDKs work in Node.js, Edge (Deno, Vercel, Cloudflare) or in the browser to directly report scores (e.g. user feedback) to Langfuse

→ SDK docs

⛓️ New Trace UI

Most users of Langfuse run complex LLM-based applications. To help monitor and debug these applications and understand how the different steps lead to the final output, we added a new nested trace UI.

Examples from our Langchain integration:

🧮 Automated Token Counts

Usage/cost reporting is a key use case of Langfuse.

Until now, token counts needed to be ingested when logging new LLM calls. For OpenAI and Anthropic models, Langfuse now automatically calculates token counts based on the ingested prompts and completions. This is helpful as some APIs do not provide token counts in their responses. It also reduces integration effort.

→ Learn more

🌟 Human-in-the-loop Evaluation

Scores in Langfuse are essential to monitor the quality of your LLM app. Until now, scores were created via the Web SDK based on user feedback (e.g. thumbs up/down, implicit user feedback) or via the API (e.g. when running model-based evals).

Many of you wanted to annotate generations in the UI as you or your team browse production logs. We’ve added this to the Langfuse UI:

→ Learn more

👥 User Reporting

Add a userId when ingesting data into Langfuse and use it to get usage/cost broken down by user. Explore what individual users are doing in your LLM app.

More user-based analytics and features are coming soon, e.g. APIs to use this data in your own dashboards and usage-based billing.

→ Learn more

📈 Analytics (alpha)

Currently we are working with a small group of alpha users to test our analytics features. We’ve focused a lot on getting this right and are excited to share more soon. Reach out if you want to share your needs or want to be part of the alpha. We are onboarding new users every week.

→ More details

👩‍💻 Public GET API

We added a public GET API to Langfuse to support users consuming Langfuse data in their own applications. Check out the API reference for more details on the available endpoints. Need other endpoints? Shoot me a message: marc@langfuse.com

Example use cases:

• Integrate user feedback (by use case) into your own dashboards
• Fine-tune on low-quality generations

GET /traces
GET /traces/:traceId
GET /scores
GET /observations/:observationId

→ API reference

📚 Improved Docs

We’ve shipped a lot over the last month and the docs were lagging behind. We’ve now updated the docs to reflect the latest changes and added a bunch of new content. This should make it way easier to get started with Langfuse and incrementally adopt more advanced features. We’d love to get your feedback on the docs, use the new widget on the bottom to 👍/👎, thank you 🙏

We also added a Q&A chatbot as one user was surprised that we don’t have one yet. It’s still a bit rough around the edges but we’ll improve it over time. Give feedback when using it and you can be sure that it is all tracked and analyzed in Langfuse.

🚢 What’s Next?

There is more coming in August. Stay tuned! We’ll focus on shipping analytics to all users and further improvements to the UI/DX of the core platform. Anything you’d like to see? Join us on Discord and share your thoughts.

Subscribe to get monthly updates via email:

Follow along on Twitter (@Langfuse, @marcklingen)