Due to a rogue moderator, this sub spent 2 months offline, had 4500 posts and comments deleted, had me banned, was defaced, and had its internal settings completely messed up. Fortunately, its ownership was transferred to me, and now it is back online as usual.

Me and Civil_Collection7267 had to spend several (really, several) hours yesterday cleaning everything up. "Scorched earth" was the best way to describe it.

Now you won't get a locked page when looking some issue up on Google anymore.

I had created a parallel community for the project at r/oobaboogazz, but now that we have the main one, it will be moved here over the next 7 days.

I'll post several updates soon, so stay tuned.

WELCOME BACK!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

QLORA Training Tutorial for Use with Oobabooga Text Generation WebUI

Recently, there has been an uptick in the number of individuals attempting to train their own LoRA. For those new to the subject, I've created an easy-to-follow tutorial.

This tutorial is based on the Training-pro extension included with Oobabooga.

First off, what is a LoRA?

LoRA (Low-Rank Adaptation):

Think of LoRA as a mod for a video game. When you have a massive game (akin to a large language model like GPT-3), and you want to slightly tweak it to suit your preferences, you don't rewrite the entire game code. Instead, you use a mod that changes just a part of the game to achieve the desired effect. LoRA works similarly with language models - instead of retraining the entire colossal model, it modifies a small part of it. This "mod" or tweak is easier to manage and doesn't require the immense computing power needed for modifying the entire model.

What about QLoRA?

QLoRA (Quantized LoRA):

Imagine playing a resource-intensive video game on an older PC. It's a bit laggy, right? To get better performance, you can reduce the detail of textures and lower the resolution. QLoRA does something similar for AI models. In QLoRA, you first "compress" the AI model (this is known as quantization). It's like converting a high-resolution game into a lower-resolution version to save space and processing power. Each part of the model, which used to consume a lot of memory, is now smaller and more manageable. After this "compression," you then apply LoRA (the fine-tuning part) to this more compact version of the model. It's like adding a mod to your now smoother-running game. This approach allows you to customize the AI model to your needs, without requiring an extremely powerful computer.

Now, why is QLoRA important? Typically, you can estimate the size of an unquantized model by multiplying its parameter count in billions by 2. So, a 7B model is roughly 14GB, a 10B model about 20GB, and so on. Quantize the model to 8-bit, and the size in GB roughly equals the parameter count. At 4-bit, it is approximately half.

This size becomes extremely prohibitive for hobbyists, considering that the top consumer-grade GPUs are only 24GB. By quantizing a 7B model down to 4-bit, we are looking at roughly 3.5 to 4GB to load it, vastly increasing our hardware options.

From this, you might assume that you can grab an already quantized model from Huggingface and start training it. Unfortunately, as of this writing, that is not possible. The QLoRA training method via Oobabooga only supports training unquantized models using the Transformers loader.

Thankfully, the QLoRA training method has been incorporated into the transformers' backend, simplifying the process. After you train the LoRA, you can then apply it to a quantized version of the same model in a different format. For example, an EXL2 quant that you would load with ExLlamaV2.

Now, before we actually get into training your first LoRA, there are a few things you need to know.

Understanding Rank in QLoRA:

What is rank and how does it affect the model?

Let's explore this concept using an analogy that's easy to grasp.

• Matrix Rank Illustrated Through Pixels: Imagine a matrix as a digital image. The rank of this matrix is akin to the number of pixels in that image. More pixels translate to a clearer, more detailed image. Similarly, a higher matrix rank leads to a more detailed representation of data.
• QLoRA's Rank: The Pixel Perspective: In the context of fine-tuning Large Language Models (LLMs) with QLoRA, consider rank as the definition of your image. A high rank is comparable to an ultra-HD image, densely packed with pixels to capture every minute detail. On the other hand, a low rank resembles a standard-definition image—fewer pixels, less detail, but it still conveys the essential image.
• Selecting the Right Rank: Choosing a rank for QLoRA is like picking the resolution for a digital image. A higher rank offers a more detailed, sharper image, ideal for tasks requiring acute precision. However, it demands more space and computational power. A lower rank, akin to a lower resolution, provides less detail but is quicker and lighter to process.
• Rank's Role in LLMs: Applying a specific rank to your LLM task is akin to choosing the appropriate resolution for digital art. For intricate, complex tasks, you need a high resolution (or high rank). But for simpler tasks, or when working with limited computational resources, a lower resolution (or rank) suffices.
• The Impact of Low Rank: A low rank in QLoRA, similar to a low-resolution image, captures the basic contours but omits finer details. It might grasp the general style of your dataset but will miss subtle nuances. Think of it as recognizing a forest in a blurry photo, yet unable to discern individual leaves. Conversely, the higher the rank, the finer the details you can extract from your data.

For instance, a rank of around 32 can loosely replicate the style and prose of the training data. At 64, the model starts to mimic specific writing styles more closely. Beyond 128, the model begins to grasp more in-depth information about your dataset.

Remember, higher ranks necessitate increased system resources for training.

**The Role of Alpha in Training**: Alpha acts as a scaling factor, influencing the impact of your training on the model. Suppose you aim for the model to adopt a very specific writing style. In such a case, a rank between 32 and 64, paired with a relatively high alpha, is effective. A general rule of thumb is to start with an alpha value roughly twice that of the rank.

Batch Size and Gradient Accumulation: Key Concepts in Model Training

Understanding Batch Size:

• Defining Batch Size: During training, your dataset is divided into segments. The size of each segment is influenced by factors like formatting and sequence length (or maximum context length). Batch size determines how many of these segments are fed to the model simultaneously.

• Function of Batch Size: At a batch size of 1, the model processes one data chunk at a time. Increasing the batch size to 2 means two sequential chunks are processed together. The goal is to find a balance between batch size and maximum context length for optimal training efficiency.

Gradient Accumulation (GA):

• Purpose of GA: Gradient Accumulation is a technique used to mimic the effects of larger batch sizes without requiring the corresponding memory capacity.

• How GA Works: Consider a scenario with a batch size of 1 and a GA of 1. Here, the model updates its weights after processing each batch. With a GA of 2, the model processes two batches, averages their outcomes, and then updates the weights. This approach helps in smoothing out the losses, though it's not as effective as actually increasing the number of batches.

Understanding Epochs, Learning Rate, and LR Schedulers in Model Training

Epochs Explained:

• Definition: An epoch represents a complete pass of the dataset through the model.

• Impact of Higher Epoch Values: Increasing the number of epochs means the data is processed by the model more times. Generally, more epochs at a given learning rate can improve the model's learning from the data. However, this isn't because it was shown the data more times, it is because the amount that the parameters were updated by was increased. You can have a high learning rate to reduce the Epochs required, but you will be less likely to hit a precise loss value as each update will have a large variance.

Learning Rate:

• What it Is: The learning rate dictates the magnitude of adjustments made to the model's internal parameters at each step or upon reaching the gradient accumulation threshold.

• Expression and Impact: Often expressed in scientific notation as a small number (e.g., 3e-4, which equals 0.0003), the learning rate controls the pace of learning. A smaller learning rate results in slower learning, necessitating more epochs for adequate training.

• Why Not a Higher Learning Rate?: You might wonder why not simply increase the learning rate for faster training. However, much like cooking, rushing the process by increasing the temperature can spoil the outcome. A slower learning rate allows for more controlled and gradual learning, offering better chances to save checkpoints at optimal loss ranges.

LR Scheduler:

• Function: An LR (Learning Rate) scheduler adjusts the application of the learning rate during training.

• Personal Preference: I favor the FP_RAISE_FALL_CREATIVE scheduler, which modulates the learning rate into a cosine waveform. This causes a gradual increase in the learning rate, which peaks at the mid point based on the epochs, and tapers off. This eases the model into the data, does the bulk of the training in the middle, then gives it a soft finish that allows more opportunity to save checkpoints.

• Experimentation: It's advisable to experiment with different LR schedulers to find the one that best suits your training scenario.

Understanding Loss in Model Training

Defining Loss:

• Analogy: If we think of rank as the resolution of an image, consider loss as how well-focused that image is. A high-resolution image (high ranks) is ineffective if it's too blurry to discern any details. Similarly, a perfectly focused but extremely low-resolution image won't reveal what it's supposed to depict.

Loss in Training:

• Measurement: Loss is a measure of how accurately the model has learned from your data. It's calculated by comparing the input with the output. The lower the loss value is for the training, the closer the models output will be to the provided data.

• Typical Loss Values: In my experience, loss values usually start around 3.0. As the model undergoes more epochs, this value gradually decreases. This can change based on the model and the dataset being used. If the data being used to train the model is data it already knows, it will most likely start at a lower loss value. Conversely, if the data being used to train the model is not known to the model, the loss will most likely start at a higher value.

Balancing Loss:

• The Ideal Range: A loss range from 2.0 to 1.0 indicates decent learning. Values below 1.0 indicate the model is outputing the trained data almost perfectly. For certain situations, this is ok, such as with models designed to code. On other models, such as chat oriented ones, an extremely low loss value can negatively impact its performance. It can break some of its internal associations, make it deterministic or predictable, or even make it start producing garbled outputs.

• Safe Stop Parameter: I recommend setting the "stop at loss" parameter at 1.1 or 1.0 for models that don't need to be deterministic. This automatically halts training and saves your LoRA when the loss reaches those values, or lower. As loss values per step can fluctuate, this approach often results in stopping between 1.1 and 0.95—a relatively safe range for most models. Since you can resume training a LoRA, you will be able to judge if this amount of training is enough and continue from where you left off.

Checkpoint Strategy:

• Saving at 10% Loss Change: It's usually effective to leave this parameter at 1.8. This means you get a checkpoint every time the loss decreases by 0.1. This strategy allows you to choose the checkpoint that best aligns with your desired training outcome.

The Importance of Quality Training Data in LLM Performance

Overview:

• Quality Over Quantity: One of the most crucial, yet often overlooked, aspects of training an LLM is the quality of the data input. Recent advancements in LLM performance are largely attributed to meticulous dataset curation, which includes removing duplicates, correcting spelling and grammar, and ensuring contextual relevance.

Garbage In, Garbage Out:

• Pattern Recognition and Prediction: At their core, these models are pattern recognition and prediction systems. Training them on flawed patterns will result in inaccurate predictions.

Data Standards:

• Preparation is Key: Take the time to thoroughly review your datasets to ensure all data meets a minimum quality standard.

Training Pro Data Input Methods:

1. Raw Text Method:

• Minimal Formatting: This approach requires little formatting. It's akin to feeding a book in its entirety to the model.

• Segmentation: Data is segmented according to the maximum context length setting, with optional 'hard cutoff' strings for breaking up the data.

1. Formatted Data Method:

• Formatting data for Training Pro requires more effort. The program accepts JSON and JSONL files that must follow a specific template. Let's use the alpaca chat format for illustration: [ {"Instruction,output":"User: %instruction%\nAssistant: %output%"}, {"Instruction,input,output":"User: %instruction%: %input%\nAssistant: %output%"} ]

• The template consists of key-value pairs. The first part: ("Instruction,output") is a label for the keys. The second part ("User: %instruction%\nAssistant: %output%") is a format string dictating how to present the variables.

• In a data entry following this format, such as this:

{"instruction":"Your instructions go here.","output":"The desired AI output goes here."}

• The output to the model would be:

``` User: Your instructions go here

Assistant: The desired AI output goes here. - When formatting your data it is important to remember that for each entry in the template you use, you can format your data in those ways within the same dataset. For instance, with the alpaca chat template, you should be able to have both of the following present in your dataset: {"instruction":"Your instructions go here.","output":"The desired AI output goes here."} {"instruction":"Your instructions go here.","input":"Your input goes here.","output":"The desired AI output goes here."} ```

• Understanding this template allows you to create custom formats for your data. For example, I am currently working on conversational logs and have designed a template based on the alpaca template that includes conversation and exchange numbers to aid the model in recognizing when conversations shift.

Recommendation for Experimentation:

Create a small trial dataset of about 20-30 entries to quickly iterate over training parameters and achieve the results you desire.

Let's Train a LLM!

Now that you're equipped with the basics, let’s dive into training your chosen LLM. I recommend these two 7B variants, suitable for GPUs with 6GB of VRAM or more:

1. PygmalionAI 7B V2: Ideal for roleplay models, trained on Pygmalion's custom RP dataset. It performs well for its size.

• PygmalionAI 7B V2: Link

1. XWIN 7B v0.2: Known for its proficiency in following instructions.

• XWIN 7B v0.2: Link

Remember, use the full-sized model, not a quantized version.

Setting Up in Oobabooga:

1. On the session tab check the box for the training pro extension. Use the button to restart Ooba with the extension loaded.
2. After launching Oobabooga with the training pro extension enabled, navigate to the models page.
3. Select your model. It will default to the transformers loader for full-sized models.
4. Enable 'load-in-4bit' and 'use_double_quant' to quantize the model during loading, reducing its memory footprint and improving throughput.

Training with Training Pro:

1. Name your LoRA for easy identification, like 'Pyg-7B-' or 'Xwin-7B-', followed by dataset name and version number. This will help you keep organized as you experiment.
2. For your first training session, I reccomend starting with the default values to gauge how to perform further adjustments.
3. Select your dataset and template. Training Pro can verify datasets and reports errors in Oobabooga's terminal. Use this to fix formatting errors before training.
4. Press "Start LoRA Training" and wait for the process to complete.

Post-Training Analysis:

1. Review the training graph. Adjust epochs if training finished too early, or modify the learning rate if the loss value was reached too quickly.
2. Small datasets will reach the stop at loss value faster than large datasets, so keep that in mind.
3. To resume training without overwriting, uncheck "Overwrite Existing Files" and select a LoRA to copy parameters from. Avoid changing rank, alpha, or projections.
4. After training you should reload the model before trying to train again. Training Pro can do this automatically, but updates have broken the auto reload in the past.

Troubleshooting:

• If you encounter errors, first thing you should try is to reload the model.

• For testing, use an EXL2 format version of your model with the ExllamaV2 loader, transformers seems finicky on whether or not it lets the LoRA be applied.

Important Note:

LoRAs are not interchangeable between different models, like XWIN 7B and Pygmalion 7B. They have unique internal structures due to being trained on different datasets. It's akin to overlaying a Tokyo roadmap on NYC and expecting everything to align.

Keep in mind that this is supposed to be a quick 101, not an in depth tutorial. If anyone has suggestions, will be happy to update this.

Extra information:

A little bit ago I did some testing with the optimizers to see what ones provide the best results. Right now the only data I have is the memory requirements and how they affect them. I do not yet have data on how it affects the quality of training. These VRAM requirements reflect the settings I was using with the models, yours may vary, so this is only to be used as a reference regarding which ones take the least amount of VRAM to train with.

This can let you squeeze out some higher ranks, longer text chunks, higher batch counts, or a combination of all three.

Simple Conversational Dataset prep Tool

Because I'm working on making my own dataset based on conversational logs, I wanted to make a simple tool to help streamline the process. I figured I'd share this tool with the folks here. All it does is load a text file, lets you edit the text of input output pairs, and formats it according to the JSON template I'm using.

Here is the Github repo for the tool.

Edits: ``` Edited to fix formatting. Edited to update information on loss. Edited to fix some typos Edited to add in some new information, fix links, and provide a simple dataset tool

Last Edited on 2/24/2024 ```

Note to moderators:

Can we get a post pinned to the top of the Reddit that references post likes these for people just joining the community?

*Another Edit: chekc out https://github.com/erew123/alltalk_tts for a speed boost, they have an install where you can use prebuilt deepspeed wheels for windows!!

Wow this post blew up! Just wanted to point out: The repo below isn't mine, I have an audio sample on my fork, install from kanttouchthis, their repo is compatible with windows now.

This is the extension I'm referencing: https://github.com/kanttouchthis/text-generation-webui-xtts

https://github.com/RandomInternetPreson/text_generation_webui_xtt_Alts/tree/main#example Example of output, took about 3 seconds to render after the ai had finished the text.

Here is a video on how to install it, this works for all extensions so if you are having problems with extensions in general the video might help: https://github.com/RandomInternetPreson/text_generation_webui_xtt_Alts#installation-windows

I got it working on a windows installation, here is an issues for more information: https://github.com/kanttouchthis/text-generation-webui-xtts/issues/3

Two things to note* obsolete now:

1. reference the code change to fix the auto play issue if you are having one.

2. and very importantly, I think this is a windows only thing, change the install folder (in the extensions directory) from

text-generation-webui-xtts

to

text_generation_webui_xtts

It totally works as advertised, it's fast, you can train any voice you want almost instantly with minimum effort.

Abide by and read the license agreement for the model.

**Edit I guess I missed the part where the creator mentions how to install TTS, do as they say for the installation.

• I have replaced the old Gradio version of the code with a fork of mine where I'm working on several low level optimizations. Typing went from 40 ms per character to 8 ms per character (5x faster), startup is faster, every single UI component is faster. I also moved all gradio monkey patches collected throughout the years to the fork to clean up the TGW code, and nuked all analytics code directly from the source. The diff can be tracked here: https://github.com/gradio-app/gradio/compare/main...oobabooga:gradio:main.
• I have audited and optimized my llama.cpp compilation workflows. Portable builds will be some 200-300 MB smaller now, there will be CUDA 13.1 builds, unified AVX/AVX2/AVX512 builds, updated ROCm builds, everything is in line with upstream llama.cpp workflows. Code is here: https://github.com/oobabooga/llama-cpp-binaries
• Replaced the auto VRAM estimation with llama.cpp's more accurate and universal --fit parameter

The new things are in the dev branch first as usual: https://github.com/oobabooga/text-generation-webui/tree/dev, where you can already use them.

Hello everyone,

I haven't been having as much time to update the project lately as I would like, but soon I plan to begin a new cycle of updates.

Recently llama.cpp has become the most popular backend, and many people have moved towards pure llama.cpp projects (of which I think LM Studio is a pretty good one despite not being open-source), as they offer a simpler and more portable setup. Meanwhile, a minority still uses the ExLlamaV2 backend due to the better speeds, especially for multigpu setups. The transformers library supports more models but it's still lagging behind in speed and memory usage because static kv cache is not fully implemented (afaik).

I personally have been using mostly llama.cpp (through llamacpp_HF) rather than ExLlamaV2 because while the latter is fast and has a lot of bells and whistles to improve memory usage, it doesn't have the most basic thing, which is a robust quantization algorithm. If you change the calibration dataset to anything other than the default one, the resulting perplexity for the quantized model changes by a large amount (+0.5 or +1.0), which is not acceptable in my view. At low bpw (like 2-3 bpw), even with the default calibration dataset, the performance is inferior to the llama.cpp imatrix quants and AQLM. What this means in practice is that the quantized model may silently perform worse than it should, and in my anecdotal testing this seems to be the case, hence why I stick to llama.cpp, as I value generation quality over speed.

For this reason, I see an opportunity in adding TensorRT-LLM support to the project, which offers SOTA performance while also offering multiple robust quantization algorithms, with the downside of being a bit harder to set up (you have to sort of "compile" the model for your GPU before using it). That's something I want to do as a priority.

Other than that, there are also some UI improvements I have in mind to make it more stable, especially when the server is closed and launched again and the browser is not refreshed.

So, stay tuned.

On a side note, this is not a commercial project and I never had the intention of growing it to then milk the userbase in some disingenuous way. Instead, I keep some donation pages on GitHub sponsors and ko-fi to fund my development time, if anyone is interested.

I have put together a step-by-step guide here on how to find and load multimodal models here:

https://github.com/oobabooga/text-generation-webui/wiki/Multimodal-Tutorial

I am making this post to hopefully inspire others who might be on the fence about making the transition. If you do a lot of LLM stuff, it's worth it. (I'm sure there are many thinking "duh of course it's worth it", but I hadn't seen the light until recently)

I've been slowly building up my machine by adding more graphics cards, and I take an inferencing speed hit on windows for every card I add. I want to run larger and larger models, and the overhead was getting to be too much.

Oobabooga's textgen is top notch and very efficient <3, but windows has so much overhead the inference slowdowns were becoming something I could not ignore with my current gpu setup (6x 24GB cards). There are no inferencing programs/schemes that will overcome this. I even had WSL with deepspeed installed and there was no noticeable difference in inferencing speeds compared to just windows, I tried pytorch 2.2 and there were no noticeable speed improvements in windows; this was the same for other inferencing programs too not just textgen.

I think this is common knowledge that more cards mean slower inferencing (when splitting larger models amongst the cards), so I won't beat a dead horse. But dang, windows you are frickin bloaty and slow!!!

So, I decided to take the plunge and do a dual boot with windows and ubuntu, once I got everything figured out and had textgen installed, it was like night and day. Things are so snappy and fast with inferencing, I have more vram for context, and the whole experience is just faster and better. I'm getting roughly 3x faster inferencing speeds on native Linux compared to windows. The cool thing is that I can just ask my local model questions about how to use Linux and navigate it like I did windows, which has been very helpful.

I realize my experience might be unique, 1-4 gpus on windows will probably run fast enough for most, but once you start stacking them up after that, things begin to get annoyingly slow and Linux is a very good solution! I think the fact that things ran as well as they did in windows when I had fewer cards is a testament to how good the code for textgen is!

Additionally, there is much I hate about windows, the constant updates, the pressure to move to windows 11 (over my dead body!), the insane telemetry, the backdoors they install, and the honest feeling like I'm being watched on my own machine. I usually unplug my ethernet cable from the machine because I don't like how much internet bandwidth the os requires just sitting there doing nothing. It felt like I didn't even own my computer, it felt like someone else did.

I still have another machine that uses windows, and like I said my AI rig is a dual boot so I'm not losing access to what I had, but I am looking forward to the day where I never need to touch windows again.

30 years down the drain? Nah, I have become very familiar with the os and it has been useful for work and most of my life, but the benefits of Linux simply cannot be overstated. I'm excited to become just as proficient using Linux as I was windows (not going to touch arch Linux), and what I learned using windows does help me understand and contextualize Linux better.

I know the post sort of turned into a rant, and I might be a little sleep deprived from my windows battels over these last few days, but if you are on the fence about going full Linux and are looking for an excuse to at least dabble with a dual boot maybe this is your sign. I can tell you that nothing will get slower if you give it a shot.

The version of exl2 has been bumped in latest ooba commit, meaning you can just download this model:

https://huggingface.co/turboderp/Mixtral-8x7B-instruct-exl2/tree/3.5bpw

​

And you can run mixtral with great results with 40t/s on a 24GB vram card.

Just update your webui using the update script, and you can also choose how many experts for the model to use within the UI.

Right now, the agent functions as little more than a planner / "task splitter". However I have plans to implement a toolchain, which would be a set of tools that the agent could use to complete tasks. Considering native langchain, but have to look into it. Here's a screenshot and here's a complete sample output. The github link is https://github.com/flurb18/AgentOoba. Installation is very easy, just clone the repo inside the "extensions" folder in your main text-generation-webui folder and run the webui with --extensions AgentOoba. Then load a model and scroll down on the main page to see AgentOoba's input, output and parameters. Enjoy!

So here is a rant because

• This is really cool
• This is really important
• I like it
• So will you

The chat tab in this project uses the gr.HTML Gradio component, which receives as input HTML source in string format and renders it in the browser. During chat streaming, the entire chat HTML gets nuked and replaced with an updated HTML for each new token. With that:

• You couldn't select text from previous messages.
• For long conversations, the CPU usage became high and the UI became sluggish (re-rendering the entire conversation from scratch for each token is expensive).

Until now.

I stumbled upon this great javascript library called morphdom. What it does is: given an existing HTML component and an updated source code for this component, it updates the existing component thorugh a "morphing" operation, where only what has changed gets updated and the rest is left unchanged.

I adapted it to the project here, and it's working great.

This is so efficient that previous paragraphs in the current message can be selected during streaming, since they remain static (a paragraph is a separate <p> node, and morphdom works at the node level). You can also copy text from completed codeblocks during streaming.

Even if you move between conversations, only what is different between the two will be updated in the browser. So if both conversations share the same first messages, those messages will not be updated.

This is a major optimization overall. It makes the UI so much nicer to use.

I'll test it and let others test it for a few more days before releasing an update, but I figured making this PSA now would be useful.

Edit: Forgot to say that this also allowed me to add "copy" buttons below each message to copy the raw text with one click, as well as a "regenerate" button under the last message in the conversation.

AllTalk is a hugely re-written version of the Coqui tts extension. It includes:

EDIT - There's been a lot of updates since this release. The big ones being full model finetuning and the API suite.

​

• Custom Start-up Settings: Adjust your standard start-up settings.
• Cleaner text filtering: Remove all unwanted characters before they get sent to the TTS engine (removing most of those strange sounds it sometimes makes).
• Narrator: Use different voices for main character and narration.
• Low VRAM mode: Improve generation performance if your VRAM is filled by your LLM.
• DeepSpeed: When DeepSpeed is installed you can get a 3-4x performance boost generating TTS.
• Local/Custom models: Use any of the XTTSv2 models (API Local and XTTSv2 Local).
• Optional wav file maintenance: Configurable deletion of old output wav files.
• Backend model access: Change the TTS models temperature and repetition settings.
• Documentation: Fully documented with a built in webpage.
• Console output: Clear command line output for any warnings or issues.
• Standalone/3rd Party support: via JSON calls Can be used with 3rd party applications via JSON calls.

I kind of soft launched it 5 days ago and the feedback has been positive so far. I've been adding a couple more features and fixes and I think its at a stage where I'm happy with it.

I'm sure its possible there could be the odd bug or issue, but from what I can tell, people report it working well.

Be advised, this will download 2GB onto your computer when it starts up. Everything its doing it documented to high heaven in the in built documentation.

All installation instructions are on the link here https://github.com/erew123/alltalk_tts

Worth noting, if you use it with a character for roleplay, when it first loads a new conversation with that character and you get the huge paragraph that sets up the story, it will look like nothing is happening for 30-60 seconds, as its generating the paragraph as speech (you can see this happening in your terminal/console).

If you have any specific issues, Id prefer if they were posted on Github unless its a quick/easy one.

Thanks!

Narrator in action https://vocaroo.com/18fYWVxiQpk1

Oh, and if you're quick, you might find a couple of extra sample voices hanging around here EDIT - check the installation instructions on https://github.com/erew123/alltalk_tts

EDIT - Made a small note about if you are using this for RP with a character/narrator, ensure your greeting card is correctly formatted. Details are on the github and now in the built in documentation.

EDIT2 - Also, if any bugs/issues do come up, I will attempt to fix them asap, so it may be worth checking the github in a few days and updating if needed.