A Shiny App for handwritten digits

Ruthger Righart

Blogs, tutorials & videos: https://rrighart.github.io

Email: rrighart@googlemail.com

Introduction

Shiny is an exciting R library for interactive statistical analyses and modeling, as well as visualisation of data, using plots, figures and tables. The user can adapt this in real time using a graphical user interface.

The current blog presents a Shiny App for handwritten digits. Handwritten digits have been used extensively to investigate automated classification of images.

This blog is not meant to be a tutorial on learning Shiny, for which the following reference is suggested [1]. Rather, it is meant to be a case example how Shiny can be used in image processing and analysis of handwritten digits, including inspection and evaluation of digits. No prior knowledge of Shiny is assumed in order to follow the instructions in this blog.

Loading the datasets

In the current context, a selection of data from the MNIST will be used. MNIST is a famous set of handwritten digits [2]. In addition, a self-made set of handwritten digits will be used [3].

To limit required space and loading time, I have created a set of 200 digits for each set, which are at my GitHub page ( https://github.com/RRighart/App-Digits ). If you would like to run the App locally on your workstation or laptop, you could download the selected data using the following code:

myurl<-"https://raw.githubusercontent.com/RRighart/App-Digits/master/selected-data.csv"
z <- tempfile()
download.file(myurl,z,mode="wb")
train1 <- read.csv(z)
file.remove(z)

## [1] TRUE

For downloading a selection of handwritten digits that I created myself, use the following code. For the current purpose the data were not preprocessed:

myurl<-"https://raw.githubusercontent.com/RRighart/App-Digits/master/selected-own-digits.csv"
z <- tempfile()
download.file(myurl,z,mode="wb")
train2 <- read.csv(z)
file.remove(z)

## [1] TRUE

Using the table function, we can see that a quite similar number of digits from 0-9 is in these selected datasets.

table(as.factor(train1[,1]))

## 
##  0  1  2  3  4  5  6  7  8  9 
## 24 24 18 20 21 17 19 19 16 22

table(as.factor(train2[,1]))

## 
##  0  1  2  3  4  5  6  7  8  9 
## 20 20 20 20 20 20 20 20 20 20

Running the App

If you do not have Shiny installed yet, you could use the code install.packages("shiny"). Then you could load the library in R by typing library(shiny):

The Shiny App can be directly run from GitHub. For that you need to type in runGitHub("<repository name>", "<username>"). Please take care that the data train1 and train2 are in your workspace (see above).

library(shiny)
runGitHub("App-Digits", "RRighart")

The App can also be run in your local space. You need to set pa, your path directory, for example pa="/home/my-directory/Shiny/". When you have set the directory, using setwd(pa), create under this directory a folder with the name App-Digits, and save the file "app.R" in this folder (the code of app.R is below or at my GitHub page).

library(shiny)
setwd(pa)
runApp("App-Digits")

About the App

When you have run the App by using the code runApp("App-Digits"), you should see the following window:

The App is a case example to demonstrate what is potentially possible with such interactive webapplications. The App has a graphic user interface in which one can scroll through images of handwritten digits by hitting the button "Next Image". The index simply indicates the rownumber of the dataset that is displayed. By selecting one of the pixels, it is possible to obtain the pixel coordinate (in the raster of 28 by 28 pixels) and pixel intensity.

On the right, the App shows a selection menu with radio buttons. The user could evaluate here which digits are displayed. This is very useful for the validation of stimuli or image files. Such validation studies are often conducted in for example experimental psychology or machine learning research. At the bottom of the window appears an option to select between different datasets, in this case the "MNIST" or "Own handwritten digits".

Code

Underneath the code that is used in the file "app.R" is shown:

library(shiny)

ui <- fluidPage(
  tags$style(type="text/css", "plot img{float:left;}"),
  
  h3("User interface for evaluating digits"),
  h6("In this interface you are requested to evaluate the label of various handwritten digits"),
  
  fluidRow(
   
    column(6, offset = 0,
      h4("Image Display"),
      plotOutput("view", height=300, click = "plot_click")
      ,
    fluidRow(  
      column(4, offset = 0,
      h6("Press button to forward"),
      actionButton("counter", label= "Next Image", icon("paper-plane"), style="color: #fff; 
      background-color: #337ab7; border-color: #2e6da4"),
      h6("Index"),
      verbatimTextOutput("value")
      ),
      column(4, offset=0,
      h6("Pixel coordinates and pixel intensity"),
      verbatimTextOutput("info")
      )
        
      )
      ),
    
    column(6, offset = 0,
      h4("Digit labels"),
      radioButtons("radio", label=NULL, choices = list("Digit 0" = 0, "Digit 1" = 1, "Digit 2" = 2, "Digit 3" = 3, "Digit 4" = 4, "Digit 5" = 5, "Digit 6" = 6, "Digit 7" = 7, "Digit 8" = 8, "Digit 9" = 9, "No Idea" = 99), selected = 0)
      ,
      
    fluidRow(  
      column(3, offset = 0,
      h6("Your selection"),
      verbatimTextOutput("value2")
      ),
      column(3, offset=0,
      h6("Select data"),
      selectInput("dataset", NA, choices = c("MNIST", "Own Digits"))
      )
      )    
      )
  ),

  fluidRow(
    
  )
)

server <- function(input, output) {

 output$value <- renderText({
    input$counter
    })

 output$value2 <- renderText({
    input$radio
    })
 
 datasetInput <- reactive({
    switch(input$dataset,
        "MNIST" = train1,
        "Own Digits" = train2)
    })
 
 output$view <- renderPlot({
    dataset <- datasetInput()
    m = t(apply(matrix(unlist(dataset[input$counter+1,-1]), nrow=28, byrow=TRUE), 2, rev))
    par(mfrow=c(1,1),
        oma = c(0,0,0,0) + 0.1,
        mar = c(2,0,2,4) + 0.1)
    image(m, col=grey.colors(255), axes=FALSE, asp=1)
    })

 output$info <- renderText({
    dataset <- datasetInput()
    x=as.numeric(input$plot_click$x)
    y=as.numeric(input$plot_click$y)
    nx=ifelse(floor((x+0.03)*28)>0, floor((x+0.03)*28), 1)
    ny=ifelse(floor((y+0.03)*28)>0, floor((y+0.03)*28), 1)
    m = t(apply(matrix(unlist(dataset[input$counter+1,-1]), nrow=28, byrow=TRUE), 2, rev))
    paste0("X coord.= ", round(nx,2), "\nY coord.= ", round(ny,2), "\nPix int.= ", round(m[nx,ny],2))
}

shinyApp(ui, server)

Closing words

This is a short example how Shiny can be used to display and collect data using an interactive interface. The interactive Shiny approach can be easily applied to any image or dataset that needs to be evaluated. Other functionalities can be added to Shiny Apps as well. Say for example that you would like to inspect handwritten digits for a certain label category (for example, the digit "7"). In Shiny it would be possible to add a menu, from which a label category from 0-9 could be selected, in order to only display digits from that category.

It should in addition be mentioned that the Shiny App can be very easily adapted to images that are used in other domains, such as crop classification, or medical image classification.

And last but not least, after data collection, Shiny can be used to adjust in real time the display parameters of tables, statistics, algorithms and plots.

Please let me know if you have questions or remarks: rrighart@googlemail.com .

References

[1]. Shiny tutorials, https://shiny.rstudio.com/

[2]. MNIST data, http://yann.lecun.com/exdb/mnist/

[3]. A blog about preprocessing your own digits, https://rrighart.github.io/Digits/