Classificiation Using Logistic Regression in Visual Basic

Introduction

Background

Super Important Terms

How does one go about turning a parameter vector and values into a hypothesis?

How does one go about turning hypotheses into predictions?

Wait a second, how do we get the values for θ?

Optimising the Optimiser

  • Record the squared partial derivatives for each feature in an array
  • .
  • Divide the learning rate for each feature by the square root of the sum of the last 10 partial derivatives.

Need even more speed?

Using the code

Load("path/to/my_file_name.csv")
Function LogisticFunction(x)
'Compute the value of the logistic function given input x
'a sigmoid function has an S shape and is given by 1/(e^(-t))
'e = 2.71... (Euler's number)

Return (1 / (1 + Math.Exp(-x)))

End Function

Function LogisticCost()

Dim cost As Double

'Loop through all the training examples
For i = 0 To exampleCount - 1

Dim exampleCost As Double

If exampleOutputs(i) = 0 Then
exampleCost = Math.Log(1 - LogisticFunction(Hypothesis(exampleInputs(i))))
Else
exampleCost = Math.Log(LogisticFunction(Hypothesis(exampleInputs(i))))
End If

cost += exampleCost

Next

'Take average
cost *= -1 / exampleCount

Return Math.Abs(cost)

End Function
Sub Minimise()

'Simple implementation of adagrad bc no way in hell am I implementing bfgs or anything that requires the hessan
'gradient = X'(h(x) - y)

Dim learningRate As New List(Of Double)
Dim derivative As New List(Of Double)
Dim pastGradientSquares As New List(Of List(Of Double))
Dim sumOfPastGradientSquares As New List(Of Double)

'Initialise all the variables
For i = 0 To featureCount
derivative.Add(1)
learningRate.Add(0.01)
sumOfPastGradientSquares.Add(0)
pastGradientSquares.Add(New List(Of Double))
Next

lastCost = LogisticCost()
deltaCost = 100

'Do this until we converge (the derivative = 0ish)
While (Math.Abs(deltaCost) > desiredDelta)

Dim difference As New List(Of Double)
Dim h As New List(Of Double)

'Reset the derivative
For i = 0 To thetas.Count
derivative(i) = 0
Next

'Loop through the hypotheses and populate the list h

For i = 0 To exampleCount - 1
h.Add(LogisticFunction(Hypothesis(exampleInputs(i))))
Next

'Get the difference

For i = 0 To exampleCount - 1
difference.Add(h(i) - exampleOutputs(i))
Next

'Multiply by the features

For i = 0 To exampleCount - 1
For j = 0 To featureCount - 1
derivative(j) += difference(i) * exampleInputs(i)(j)

'Update the list of previous squared derivatives
pastGradientSquares(j).Add(Math.Pow(derivative(j), 2))

'If we exceed 10 things, just remove the oldest entry
If pastGradientSquares(j).Count > 10 Then
pastGradientSquares(j).RemoveAt(0)
End If

'Update the sums
sumOfPastGradientSquares(j) = Sum(pastGradientSquares(j))

Next
Next

'Multiply by Learning Rate for this specific feature and get new thetas

For i = 0 To featureCount - 1
thetas(i) -= (learningRate(i) / (Math.Sqrt(sumOfPastGradientSquares(i) + 0.00000001))) * derivative(i)
Next

Dim currentCost = LogisticCost()
deltaCost = currentCost - lastCost

'We need to look like we're doing something so here's something to keep users occupied
Console.WriteLine("Training... " + LogisticCost().ToString + " " + derivative(0).ToString) ' + " " + derivative(1).ToString + " " + derivative(2).ToString)

lastCost = currentCost

End While

End Sub
Function Predict(x)

Return LogisticFunction(Hypothesis(x))

End Function
Sub Check()

Dim correctlyClassified As Double = 0

'Loop through results and see how many were classified correctly
For i = 0 To exampleInputs.Count - 1

Dim result As String
If Predict(exampleInputs(i)) > 0.5 Then
result = "Positive"
If exampleOutputs(i) = 1 Then
correctlyClassified += 1
End If
Else
result = "Negative"
If exampleOutputs(i) = 0 Then
correctlyClassified += 1
End If
End If

Dim trueClassification As String
If exampleOutputs(i) = 1 Then
trueClassification = "Positive"
Else
trueClassification = "Negative"
End If

Console.WriteLine(result + " and was actually " + trueClassification)

Next

Console.WriteLine("Correctly Classified " + correctlyClassified.ToString + " out of " + exampleInputs.Count.ToString)

End Sub

Demo

Other Things to Look Into

Reguarlisation

Multiclass classification

Source

History

- Peter the Mentor, age 16

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Acorn Aspirations

Acorn Aspirations

Powering the Next Generation of Thought Leaders, Innovators and Technologists in #AI @teensinai #TeensInAI #GirlsinAI #MCStartup2016 Founder @elenasinel