Making Sense of Big Data
How does Datashader work and why is it crazy fast?
A few months ago, I wrote an article on my favorite Python Viz tools — HoloViz. Many people are interested in learning more about Dashshader — the big data visualization tool in the HoloViz family. I absolutely love Datashader and love how Datashader creates meaningful visualizations of large datasets very…
Hands-on Tutorials
Stratification, CUPED, Variance-Weighted Estimators, and ML-based methods CUPAC and MLRATE
Why do we need variance reduction?
When we do online experiments or A/B testing, we need to ensure our test has high statistical power so that we have a high probability to find the experimental effect if it does exist. What are the factors that might affect power? …
My favorite Python Viz tools — HoloViz
It is surprising to me that many data scientists do not know HoloViz. HoloViz is my favorite Python viz ecosystem, which comprises seven Python libraries — Panel, hvPlot, HoloViews, GeoViews, Datashader, Param, and Colorcet.
Why do I love Holoviz?
HoloViz allows users to build Python visualization and interactive dashboard with super easy and flexible Python…
Hands-on Tutorials
Streaming and Refreshing
Data scientists use data visualization to communicate data and generate insights. It’s essential for data scientists to know how to create meaningful visualization dashboards, especially real-time dashboards. This article talks about two ways to get your real-time dashboard in Python:
- First, we use streaming data and create an auto-updated streaming…
Minimal effort to make slides and host an html file on Github
Check out the slideshow of this article here: https://sophiamyang.github.io/slides_github_pages/.
There are two parts to this article:
- How to turn your Jupyter Notebooks into a slideshow and output to an html file.
- How to host an html file on Github.
Jupyter Notebook slides
First, let’s create a new environment slideshow, install a Jupyter notebook…
Intake driver for salesforce
A Salesforce database can be a hot mess. The figure below illustrates the relationship among some of the data tables in Salesforce. As you can see, the relationship among data tables (i.e., objects) can be complicated and hard to work with. I wrote a blog post previously on how to…
SQLAlchemy, Python Client for Google BigQuery, and bq command-line tool
How do you query BigQuery data? This article talks about 3 ways to query BigQuery data in Python. Hope you find them useful.
Environment
conda install notebook google-cloud-bigquery sqlalchemy pybigqueryCredentials
To authenticate Google Cloud locally, you will need to install Google Cloud SDK and log in/authenticate through the following command line…
Math and gradient descent implementation in Python
Multiclass logistic regression is also called multinomial logistic regression and softmax regression. It is used when we want to predict more than 2 classes. A lot of people use multiclass logistic regression all the time, but don’t really know how it works. …
Using pytest and hypothesis for unit testing
Software testing is essential for software development. It is recommended for software engineers to use test-driven development (TDD), which is a software development process that develops test cases first and then develops the software. For data scientists, it is not always easy and plausible to write tests first. Nevertheless, software…
setup, debug, version control, and deployment
Many data scientists like to use Jupyter Notebook or JupyterLab to do their data explorations, visualizations, and model building. I know some data scientists refuse to use Jupyter Notebook. But, I love to use Jupyter Notebook/Lab to do my experiments and explorations. …
