Machine Learning at the Intersection of Artificial Intelligence and Data Mining

Categories: Tech | Pubby Cash Received:  10

Artificial Intelligence (AI), Machine Learning (ML), and Data Mining (DM) are very fancy words nowadays in technology fields. But do you know the differences between these terms? They are all dedicated to the transformation of data to usable knowledge and support decisions. In other words, almost all disciplines will need them. AI is a broad term for using either data or knowledge to offer solutions to existing problems that require some search or reasoning. ML is a specific subarea of AI that offers procedures learning from data. DM, is the an interdisciplinary field to discover patterns in large data sets and subsumes both ML, statistical modeling, or visualization disciplines. The following figure gives you an idea about the relationship of the three terminologies, and the bracketed numbers in the figure denote the number of papers dealing with these technologies in the water and wastewater sectors, respectively, according to Hadjimichael et al. in a paper published in 2019. With regard to the specific methods within the water and wastewater fields, artificial neural network is the most popular method followed by clustering. There are huge potentials for further exploration, as the encapsulation of ML into functional decision support system is not fully investigated. This is because: (1) there is a lack of an association between the fields of water engineering and computer engineering; (2) most ML techniques were adopted by academics with limited practical experience; (3) ML techniques are inherently multi-disciplinary and combined with problems of great complexity facing water and wastewater systems; and (4) there is a lack of user-friendly interfaces and support after product delivery. With that said, I'm working on an integration of machine learning into my discipline of water and wastewater treatment. Sound promising?...  Read more

Deployment of Writing Static html Pages to the Web Server

Categories: Tech | Pubby Cash Received:  10

In the previous post titled "How to Make Static .html Pages Accessible with Flask in Python", we've solved the obstacle of html accessibility. Now the question becomes: how to make those static html pages? The basic idea is to query the database to generate these html pages and then save these html pages to the static "html" folder. To do this, we will need to use the BeautifulSoup module I've introduced in the an earlier post. First, we try to request the pages from the web server one at a time based on the article id of each page. Then we will parse the response from the server with the BeautifulSoup module. Since the result is the BeautifulSoup class, we will need to convert it to a string, and then write the string into a html file. The tricky parts are: (1) how to write to a file if the file does not exit? (2) how to avoid the character code error (e.g. a gbk code error might be thrown out) during the writing? and (3) how to make the process efficient by not re-writing the files again and again? For the first part, I could use with open('file_name', 'w') as file:     file.write() For the second part, however, the above code will not work. To solve the character code error, I will need to import the io module, and re-write the above code like this: with io.open('file_name', 'w', encoding='utf-8') as file:     file.write() For the third part, I let a script auto-run in the background with a nested loop. In the inner loop, I query the most recent posts published during the week and only write those new files. In the outer loop, such queries and writings are controlled to be run once a week. Want to see a showcase? Access my last post via this static html page! http://gopubby.com/html/173.html...  Read more

How to Make Static .html Pages Accessible with Flask in Python

Categories: Tech | Pubby Cash Received:  10

The following code can easily start a development web server on your machine with the flask module: from flask import Flask app = Flask(__name__) if __name__ == '__main__':     app.run(debug=True) Now, you would like to make any .html static files accessible using URL routing. What should you do? Here are the steps you need to follow:
Step 1: The above code needs to be revised like this: from flask import Flask, render_template app = Flask(__name__) @app.route('/html/<string:page_name>/') def render_static(page_name):     return render_template('html/%s.html' % page_name) if __name__ == '__main__':     app.run(debug=True) Step 2: Now Flask will look for any .html files in the "templates/html" folder. Go ahead to create a folder named templates in the root, and then create a subfolder named "html" within the "templates" folder. Step 3. Transfer all your static .html files into this html folder. And then all these files are accessible by the URL http://xxx.com/html/file_name Note: You cannot directly create an html folder in the root. Otherwise, the web server cannot be started, but throw out an error "ModuleNotFoundError: No module named 'htmlentitydefs'". If you delete the folder named html in the root, the error will be gone. This is because the folder name tricked Flask. By default, Flask will look into the "templates" folder for html pages. If you try to place your .html pages into a folder named other than "templates", it won't work either. Make sure be aware of that! If you are still not satisfied with this method, because with this method, it still cannot show a URL ending with a physical .html page. As you can see in the link above, the .html file name extension is cut off in the URL. To save the .html file extension in the URL the "send_from_directory" module will do the trick. from flask import Flask, render_template, send_from_directory app = Flask(__name__, static_url_path='') @app.route('/html/<path:path>') def send_html(path):     return send_from_directory('html', path) if __name__ == '__main__':     app.run(debug=True) Create a folder named 'html' in the root, and place all static .html pages into that folder. Those files will be accessible and the full name with .html extension will also be displayed in the URL like this: http://xxx.com/html/file_name.html Now you can name a folder "html" in the root in this case. That's why I like the second solution better. ...  Read more

Shape of a Column in Pandas Data Frame

Categories: Tech | Pubby Cash Received:  10

Pandas is a must-have Python module for data manipulation and analysis. It can read a csv file into Python in a format of pandas data frame. Let's talk about the shape of a data-frame column, which may confuse a lot of beginners. Let's say we have a csv table named "data.csv" like this:

Year	Counts
2008	100
2009	200
2010	300
2011	400
2012	500

After reading the csv file with pandas by executing the following code:
import pandas as pd
df = pd.read_csv('data.csv')
df.set_index('Year', inplace=True)
y = df['Counts']
print(y, type(y), y.shape)

You will get a variable y with two columns - Year as index and the Counts. Not one column as one may think! The type of y belongs to a class of pands.core.series.Series and the shape of the y is (5, ). Note this is zero dimension. This is what confuses people most. How can an object of two columns has a shape of zero dimension? What should we do if we only want one column - 'Counts'?

Let's execute the following codes to see the results:
y = df['Counts'].tolist()
print(y, type(y))

Now we got a list with only 'Counts' values. The data type is list. Since it is a list, the shape method cannot be used.

Then, let's execute the following codes to see the results:
y = df['Counts'].to_numpy()
print(y, type(y),y.shape)

It appears we got a list with only 'Counts' values. But this list is not a true list, as the shape method can be applied. It is a numpy array with a shape of (5, ), meaning zero dimension. Note that although we did not import numpy here, pandas is built on numpy and thus possess numpy's array features.

This zero dimension can cause some problems if you want to apply some machine learning algorithms to the data using the sci-kit learn module, because it accepts numpy array with at least one dimension. The question becomes: how to change a zero dimension array to a one-dimension array? A reshape(-1,1) method can be neat here. Execute the following code and see the results:
y = df['Counts'].to_numpy().reshape(-1,1)
print(y,type(y),y.shape)

You will see this time, y has a shape of (5,1), although you cannot really visually tell the difference compared to the previous y. But it is now a one-dimensional numpy array that can be executed with machine learning algorithms!

Although you don't have to go through the processes like these to plot the results with matplotlib, you have more flexibility and more control over your x and y axis, which is a good thing. Here is an example I did with my data:

...  Read more

Matplotlib - A Fantastic Plotting Module in Python

Categories: Tech | Pubby Cash Received:  20

I have been using a variety of plotting software for research and engineering drawing purposes, which include excel spreadsheet, sigmaplot, origin, minitab, matlab, and autoCAD. While studying matplotlib recently, I find it is time to switch to this module with python. It is so powerful that all plot details can be customized. For my applications, I need a plotting tool with the following features: (1) Linear regression. Matplotlib overkills it by possessing machine learning algorithms with a scikit-learn module. For linear regression, use from sklearn.linear_model import LinearRegression (2) Dual axis (3) Multiple plots in one figure; the canvas size of the figure can be defined; the canvas size of each plot in it can be defined as well. (4) Axis break (5) Customized error bars (6) Easy formatting of fonts, legend, axis scales, ticks and labels. With coding, no hand-formatting is needed like I did in excel spreadsheet before. (7) Easy labeling with text, annotating with arrows, and drawing additional horizontal and vertical lines (8) Inclusion of superscript, subscript, and special characters such as greek letter mu in axis labels (9) Easy export as high DPI figure while the file size is still small. This is perfect for journal submissions. The following plot is a sample I just created as a showcase how powerful it is to satisfy my above-mentioned needs. ...  Read more