Regent Rag

Regent Rag is a collection of Python scripts designed to scrape websites behind cookie-based authentication, transform the textual content into a vector database, and execute queries against this database.

This README provides instructions on how to install, setup and run the scripts locally.

Installation on Windows can be found under docs/WINDOWS.md. Note that the instructions require Windows 10 OS Build 2004 or later.

Install Python 3.11.6. You can download it from the official Python website or if you're on MacOS using pyenv.

brew install pyenv
pyenv install 3.11.6

Point out your pyenv path in ~/.zshrc

# In terminal
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

# or in shell file add:

# python env
export PYENV_ROOT=$HOME/.pyenv
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# reload shell (~/.profile, ~/.bash_profile or ~/.zshrc), e.g. .zshrc
source ~/.zshrc
# ensure it's there
cat ~/.zshrc

Install pipenv using the following command:

pip install --user pipenv

# or with brew

brew install pipenv

Now ensure pipenv is available on PATH by running:

pipenv --version

If it's not available you likely need to add ~/.local/bin to your PATH:

export PATH="$HOME/.local/bin:$PATH"

and to persist it, add it to your profile of choice (~/.profile, ~/.bash_profile or ~/.zshrc)

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc

Tell pipenv to install virtual environments in project and ensure the LANG environment variable is set:

export PIPENV_VENV_IN_PROJECT=1 && export LANG="en_US.UTF-8"

or to persist:

echo 'export PIPENV_VENV_IN_PROJECT=1\nexport LANG="en_US.UTF-8"' >> ~/.zshrc

You can verify your installation by running:

pipenv --version

Ensure you have a rust compiler installed. It's required for volta. You can install it (on MacOS) using the following command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

or with Brew

brew install rust

You can verify your installation by running:

cargo --version

https://volta.sh/ is a universal JavaScript Tool Manager (replaces nvm for Node version management)

curl https://get.volta.sh | bash

volta install node@18

Install all packages (including dev dependencies) using the following command:

make install-dev

If you are able to run the following set of commands, you should be all set!

python --version &&
pipenv --version &&
cargo --version &&
node --version &&
volta --version &&
echo $PIPENV_VENV_IN_PROJECT &&
echo $LANG

Create a .env file with the keys listed in .env.sample:

OPENAI_API_KEY=""
PINECONE_API_KEY=""
PINECONE_INDEX_NAME=""
PINECONE_ENVIRONMENT="gcp-starter"
LOG_LEVEL="INFO"

• OPENAI_API_KEY: This is your API key for OpenAI. It's used to authenticate your application with OpenAI's services. You can find them here: https://platform.openai.com/account/api-keys

• PINECONE_API_KEY: This is your API key for Pinecone. It's used to authenticate your application with Pinecone's services.

• PINECONE_INDEX_NAME: This is the name of the index in Pinecone where your vectors will be stored.

• PINECONE_ENVIRONMENT: Used to specify the Pinecone environment in which your application is running. If you are using the free tier it should probably be set to "gcp-starter".

• LOG_LEVEL: This sets the level of logging for the application. It can be set to different levels like "INFO", "DEBUG", "ERROR", etc. depending on the amount of information you want to log. "INFO" is a standard level that will capture most of the important information without being too verbose.

You need to copy a request from the network tab of your browser in order to be able to scrape a webpage that requires authentication.

These instructions may differ from browser to browser and OS to OS. It is very important that the output you paste into request.curl follows the exact format specified in request.curl.sample.

For example, if on Windows and using Firefox, I'd recommend choosing Copy Value -> Copy Request Headers and then manually fixing the lines to match the format in the sample file.

1. Open the webpage you want to scrape in Google Chrome.
2. Press F12 to open the Developer Tools.
3. Click on the Network tab.
4. Refresh the webpage to start capturing the requests.
5. Look for a request towards the domain of the webpage you want to scrape.
6. Make sure the request contains the necessary cookies for authentication. These are usually found in the Headers section of the request.
7. Right-click on the request and select Copy -> Copy as cURL.
8. Paste the copied request into a new file and save it as request.curl.

By following these steps, you'll have a request.curl file that contains all the necessary cookies for the scraper to work with authentication.

make scrape

make splits

make embeddings

make retrieval

make flask

https://github.com/Sstobo/Site-Sn33k