PagedAttention and vLLM Explained: What Are They?

4 Jan 2025

This paper proposes PagedAttention, a new attention algorithm that allows attention keys and values to be stored in non-contiguous paged memory

General Model Serving Systems and Memory Optimizations Explained

4 Jan 2025

Model serving has been an active area of research in recent years, with numerous systems proposed to tackle diverse aspects of deep learning model deployment.

Applying the Virtual Memory and Paging Technique: A Discussion

4 Jan 2025

The idea of virtual memory and paging is effective for managing the KV cache in LLM serving because the workload requires dynamic memory allocation

Evaluating vLLM's Design Choices With Ablation Experiments

4 Jan 2025

In this section, we study various aspects of vLLM and evaluate the design choices we make with ablation experiments.

How We Implemented a Chatbot Into Our LLM

4 Jan 2025

To implement a chatbot, we let the model generate a response by concatenating the chatting history and the last user query into a prompt.

How Effective is vLLM When a Prefix Is Thrown Into the Mix?

4 Jan 2025

We explore the effectiveness of vLLM for the case a prefix is shared among different input prompts

How Good Is PagedAttention at Memory Sharing?

31 Dec 2024

We evaluate the effectiveness of memory sharing in PagedAttention with two popular sampling methods: parallel sampling and beam search.

LLaVA-Phi: Limitations and What You Can Expect in the Future

29 Dec 2024

We introduce LLaVA-Phi, a vision language assistant developed using the compact language model Phi-2.

LLaVA-Phi: Qualitative Results - Take A Look At Its Remarkable Generelization Capabilities

29 Dec 2024

We present several examples that demonstrate the remarkable generalization capabilities of LLaVA-Phi, comparing its outputs with those of the LLaVA-1.5-13B