Large language models have substantially advanced the state of the art in various AI tasks, such as natural language understanding and text generation, and image processing, multimodal modeling. In this talk, I am going to talk about how we build GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model at least as good as GPT-3 and unveil how models of such a scale can be successfully pre-trained. Based on GLM-130B, we have developed ChatGLM, an alternative to ChatGPT. A small version, ChatGLM-6B, is opened with weights and codes. It can be deployed with one RTX 2080 Ti (11G) GPU, which makes it possible for everyone to deploy a ChatGPT! It has attracted over 1,000,000 downloads on Hugging Face in one month, and won the trending \#1 model for two weeks. \newline GLM-130B: \url{https://github.com/THUDM/GLM-130B} \newline ChatGLM: \url{https://github.com/THUDM/ChatGLM-6B}