Development Framework for the Evaluation of Usability in E-Government: A Case Study of E-Finance Government of Malang

. E-government is an effort to utilize information and communication technology especially internet to improve public service quality which generally implemented in a web based application. Usability is one of the important quality criteria for the success of a web. In this study we developed a framework for evaluation of usability in e-government consisting of eight stages: (1) determining the evaluation objectives, (2) determining the usability aspects, (3) determining the metrics usability, (4) selecting usability evaluation method candidates, (5) determining the required criteria of the method to be evaluated, (6) evaluating the method, (7) selecting and making the instrument, and (8) evaluate usability. . The results of the application of this framework in the case study of e-finance resulted in two methods used: user testing and questionnaires. The evaluation of usability in e-government for e-finance case studies using the proposed framework results in usability level of e-finance in terms of effectiveness, efficiency, and user satisfaction are 96%, 92%, and 70 respectively. Which can be identified to be grouped into 16 problems consisting of aspects of effectiveness and efficiency.

1 Introduction E-government is the use of information and communication technology by the government, particularly internet web based application, regarding provide convenient of information access and government service to the people, to improve service quality and to provide opportunity for people to participate in the democration process and the intitution [1]. E-government operation in order to improve access toward information and government services significantly increases, this may incline the efficiency of the site, thus it can be employed as a more effective tool for government transactions [2]. However, e-government focuses more on technology compared to user desires [3], hence it potentially causing problems in user performance and satisfaction. The problems that arise may cause the main failure of the implementation of e-government usage in the community. the amount of time, the number of steps, the completion of the task, the completeness of the content, and the satisfaction of the respondents.
Karunasena & Deng [11] developed a framework for evaluating the public value of e-government from the point of view of citizens. The resulting framework consists of four dimensions, each dimension consisting of attributes used to measure the performance of e-government in each dimension.
Hasan, Morris, & Probets [12] developed the framework for usability evaluation in different contexts of e-commerce. The framework consists of four stages: (1) using google analytics tool, (2) determining web metrics, (3) defining usability evaluation methods ie user testing and heuristic evaluation, and (4) redesigning. This research combines three methods: user testing, heuristic evaluation and google analytic tool to evaluate and identify usability problems on e-commerce website. In the research it can be seen that the steps taken for usability evaluation is clear and what method will be used.
Merwe & Bekker Research [13] developed a framework and criteria for the evaluation of e-commerce websites. The framework developed for the evaluation of ecommerce websites is mapped to three different levels. The framework can be used to identify the performance of the website and which parts can be improved.
Based on several previous research coonducted, indicating that many frameworks have been developed for evaluation of usability. But the research aimed at making the framework of e-government usability evaluation still not much done and those research were only focused to provide parameters measured when evaluating usability on egovernment and can only identify problems in e-government. Hence, the authors in this study will design a framework that consists of what steps should be performed and the parameters generated from each step. In addition, the designed framework not only able to identify problems, but also able to measure the usability level of the system. The proposed framework is expected to be used as a workflow during evaluating usability on e-government. The results of this usability evaluation are expected to be used as recommendations for improvements toward e-government.

Framework Development
This section describes the steps that will be taken to arrage the framework. The first step to create a framework is conducting the literature review which consist of evaluation objectives, e-government phases, usability aspects, and evaluation criteria and strategies.

Evaluation Objectives
Evaluation has three main objectives: to assess the level and accessibility of system functionality, to assess user interaction experiences, and to identify system-specific problems [14]. The function of the system is very important, wherefore it must be in accordance with the needs of the user to make it easier when completes tasks using the system.
The purpose of the system design evaluation is to assess the interaction of the user experience relates to how simple the system to be understood, the usefulness of the system, and user satisfaction with the system, while the identification of design-specific issues related to the system design if it it usage will cause confusion among users.

Usability Aspect
There is several view points about usability definition. According to Nielsen [15] , usability is a quality attribute that assesses how simple user interface is used. Usability also refers to method in term to improve user convenience during the design process. Based on the definition above, usability consists of 5 components of quality as follows (1) learnability (how easily users complete the basic tasks assigned to them for the first time), (2) efficiency (quality to assess how fast the user can complete tasks that have been given), (3) memorability (quality to assess how easy the user can rebuild the ability to use the system, when they reuse the system after a period of not using it), (4) errors (quality to assess how many errors generated by users, how fatal the errors, and how they able to fix those mistakes easily, (5) satisfaction (quality to assess how much the pleasure user get of the system used).
Usability is also defined as the measurement about the extent the product can be used by a particular user to achieve a particular goal with effectiveness, efficiency, and satisfaction in the context of a particular usage [8]. The definition generates 3 attributes: (1) effectiveness (accuracy and completeness achieved by the user for a particular purpose), (3) efficiency (resources used in relation to the accuracy and completeness the user uses to achieve the goal), (3) satisfaction (being free of discomfort and a positive attitude toward the product).
Based on several view points mentioned above, hence usability aspects that can be employed during usability evaluation to e-government are effectivness, efficiency, satisfaction, learnability, memorability, and error.
When evaluating the usability, despite to determine aspects of usability to be used when evaluating usability, it also needs usability metrics. Usability metrics can be used to measure the usability of a system. According to Sauro & Kindlund [18] there are 4 metrics in quantitative usability models: task time, error, task completion, and avg. Satisfaction. The four metrics are three aspects of usability according to ISO 9241-11 (effectiveness, efficiency and user satisfaction).
Learnability is how convenient users are able to complete the basic tasks assigned to them for the first time. The process of collecting and measuring the aspects of learnability was conducted by gathering the data several times. Each data collection is known as the trial. Learnability can be measured by calculating the ratio of learningability by calculating the ratio of first and second experiments [16].
Measuring the memorability of an application can be performed through calculating the number of mouse clicks until the task is completed, the time used to complete the task, the number of pages and steps used to complete the tasks [17]. The calculation is being done if user performs a task on the test and then performs the re-calculation after some time has been set not to engage the application. It's to measure how users recall and retain their capabilities.

Evaluation and Strategy Criteria
Before selecting the proper evaluation method, it has to point out the criteria that will be utilized as the ground when choosing usability evaluation method. The steps taken in order to choose usability evaluation method as follows [18]: a. Selecting a set of candidate method to be evaluated At this stage, the selection of candidate evaluation method will be evaluated. Usability evaluation methods (UEMs) are procedures that comprise of a well-defined set of activities to collect usage data related to end-user interactions with software products and how the specific nature of the software product contributes to a certain usability level [19].
Paz and Pow-Sang [20] conducted a systematic literature review to identify the evaluation methods used over the past three years to assess the usability levels of the software. The results obtained are the five most commonly used methods, namely usability testing, questionnaire, heuristic evaluation, thingking aloud, and interview. Another research is about systematic mapping study to identify the most commonly used usability evaluation method in software development [21]. The study results indicate that the five most frequently used methods of usability evaluation of web application type are questionnaire, user testing, heuristic evaluation, interview, and user testing thingking aloud. Based on previous research, authors considered 5 usability evaluation methods as candidates to be evaluated, which are questionnaire, user testing, heuristic evaluation, interview, and user testing thingking aloud.
b. Determining necessary criteria of evaluated method This stage was conducted criteria selection to be utilized in order to assess usability evaluation method candidate. The criteria selected at this stage comprises of two categories: the usability aspect to be measured upon the system and the characteristics of the usability evaluation method. The selected usability aspect is adjusted to the aspect used when perform usability evaluation. The second criteria is to select the characteristics of the usability evaluation method that can be engaged evaluating the method. There is three characteristics used, as follows: effectiveness (the ability of usability evaluation method in term of evaluating usability aspect of a system), cost (cost spent when applying usability evaluation method to detect usability issues) and repeatability (a potency to implement usability evaluation method toward several context of evaluation) [18].
Based on criteria mentioned above, thus authors engaged 3 characteristics that would be utilized to assess usability evaluation method are cost, effectiveness, and repetability.
c. Conducting Method Evaluation This stage involved two steps, the first step was based on author's assessment of probability of each usability evaluation method in term of evaluateing every aspect of usability used. At this stage the assessment performed is subjective, since there is no empirical data about the usability evaluation method to evaluate usability aspects. The scoring for the assessment level of each usability evaluation method was using three ordinal scales, which were hight (H), low (L), and medium (M Ordinal scale condition from quetions above is if the answer of Q1= "yes" and Q2= "yes", thus the socer is "H". If the answer of Q1= "yes" and Q2= "no", thus the score is "M". If the answer Q1= "no", thus the score is "L." The second stage was to conduct a more detailed analysis of candidate usability evaluation method based on 3 basic characteristics of usability evaluation method, namely cost, effectiveness, and repeatability. Cost and repeatability characteristics employ 3 ordinal scales of high (H), medium (M), and low (L). Factors that determine the cost included the number of users needed, the required infrastructure, time spent for data collection and analysis. Repeatability characteristics were assessed based on the availability of guidelines and evaluation instruments of each usability evaluation method, whereas effectiveness is assessed from the number of usability aspects that obtain H and M values as a proportion of the number of usability aspects.

The Proposed Framework
This section illustrates the proposed framework based on the literature review described in the previous section. The proposed framework is built of literature studies that had been carried out in the previous section. The proposed framework that had been developed can be seen in the Figure 1. In Figure 1 we may notice that the framework consists of eight steps as follows: (1) determining the evaluation objectives, (2) determining the usability aspects, (3) determining the metrics usability, (4) selecting usability evaluation method candidates, (5) determining the required criteria of the method to be evaluated, (6) evaluating the method, (7) selecting and making the instrument, and (8) evaluate usability.

Case Study
In the section describes about the implementation of framework that had been developed in the e-finance case study. Data collection was conducted on 5 users at the Local Device Work Unit in urban area of Malang as the treasurer of expenditure. In every kelurahan, a single person was taken as the user to perform usability evaluation of e-finance. Meanwhile, when performed usability testing that had been employed, 5 participants can reveal about 80% of all usability issues, which exist in the product [25]. If the number of users is added, then the difference in the number of problems obtained is not significant, as there will be similarity problems found among other users.

Determining evaluation subjectives
The first stage of performing e-government usability evaluation of e-finance case studies is to determine the purpose of the evaluation. This aims to facilitate the selection of methods that will be used to evaluate the usability of e-government case studies of e-finance. The case study of e-finance, the authors consider to conduct usability evaluation is to assess the level and system accessibility of functionality and identify the specific issues in the system. This objective was gained through the primary results of the previous research that had been conducted, which was assessing the level and system accessibility of functionality in term to get value on the aspect of ease of use of 74.73 [26].

Determining usability aspect
E-finance is one of the implementation of e-government in the form of web-based applications used to perform financial transactions online in Malang city government. When evaluating usability on the e-finance aspect used is effectiveness, efficiency and user satisfaction. Aspects of learnability and memorability are not used to evaluate usability on e-finance because based on surveys conducted in urban villages, the use of e-finance to transact online is used routinely every month for financial reporting. It makes learningability impossible for measurement since the previous e-finance user has already operated the application. The analysis is in accordance with the definition of learnability that is how easy the user completes the basic tasks assigned to them for the first time. The memorability aspect also does not allow for measurements on the usability evaluation of e-finance. This is because memorability is a quality to judge how easily users can rebuild the ability to use the system, when they reuse the system after a period of not using it. Based on these definitions then when performing memorability measurement is done twice that experiment 1 is done and experiment 2 is done again with span of few days and user does not use the application. This is not possible if applied when evaluating usability on e-finance because the application is used at any time in realtime when doing financial transactions. Based on the analysis and consideration, the authors chose three aspects of usability, namely effectiveness, efficiency and user satisfaction.

Determining usability metrics
Right after selecting aspect to be measured during usability process upon egovernment of e-finance case study, then the next step is identify usability metric of every single aspect used. According to the determined usability aspect on previous stage, there were four usability metrics used namely completion level, error, completion time, and user satisfaction average rate.

Determining usability evaluation method candidate
The next phase is selecting usability evaluation method candidate. This framework provides 5 method candidates that mostly used regarding web application type, namely questionnaire, user testing, heuristic evaluation, interview, and user thinking aloud. Those five usability evaluation method candidates will be selected one only to perform usability evaluation.

Determining necessary criteria needed based on method evaluated
This stage was engaged criteria determination to perform usability evaluation method candidate. This framework has two types that would be used, namely usability aspect based criteria to conduct usability evaluation and usability evaluation method characteristics, which are cost, effectiveness, and repeatability.

Performing method evaluation
This stage is carried out to recognize evaluation method used. There were two steps conducted to evaluate usability evaluation method candidate, as follows: a. Performing usability aspect evaluation toward usability evaluation method candidate This initial step assessed usability aspect toward usability evaluation method. When perform usability aspect measurement upon usability evaluation method have to adjust with evaluation subjective used. This stage uses subjective assessment, as there is no empirical data toward usability evaluation method usage in order to evaluate usability aspect. b. Performing usability evaluation candidate analysis to usability evaluation method charactersitics The second step is a more detail analysis about usability evaluation candidate based on 3 usability evaluation method basic characteristics, namely cost, effectiveness, and repetability. Method evaluation analysis results can been on table 1 and 2.
Based on analysis results on table 1 and 2, thus the selected method regarding usability evaluation is quetionnaire and user testing. Whereas, those two methods are reciprocally, user testing can be utilized to identify issue and measure usability level of effectiveness and efficiency aspects, while quetionnaire can be worthwile to measure user satisfaction rate, which cannot be completed through user testing.

Selecting and Arraging Instrument
Once after selecting the proper method associated to goals and aspects used, the next step is selecting and developing the instrument. The method used to evaluate usability of e-government of e-finance case study is user testing and quetionnaire. Below is the instrument used of every method: a. User Testing User testing instrument used is composed of a set of task that consist of 10 tasks to be completed by the user during usability evaluation of e-finance. Below is an example of the task: Task 2: Make SPP LS for honorarium security guard March 2017! b. Questionnaire This method was used to assess satisfaction attribute. The questionnaire used was Software Usability Scale (SUS). SUS is the questionnaire comprises of 10 items with 5 Likert Scale. Odd item numbers have positive statements and even items have negative statements.

Performing usability evaluation
This stage performed usability evaluation of e-finance by engaging elected method, namely user testing and questionnaire. Results data collection will be analyzed to recognize usability level and identify e-finance issues.
Once usability evaluation of e-finance is done using recommended framework, then data was analyzed to determine usability level and existing problems of e-finance application.

User testing
This method is engaged to analyse usability level and identify e-finance issue in effectiveness and efficiency aspects.
1) Effectiveness User effectiveness measurement towards e-finance was using two metrics, namely completion level and error. Below is effectiveness aspect measurement: a. Task Completion Task completion is measured by completion level assessment on every task of usability testing conducted. This metric has two values, namely task success and task failure. Table 3 presents completion level of each task. Table 3. Task Completion Level Using E-finance Task   T1  T2  T3  T4  T5  T6  T7  T8  T9  T10 Partisipants P1   Task   T1  T2  T3  T4  T5  T6  T7  T8  T9   On Table 3, it can be seen that completion task level for task 1, 2, 3, 6, 7, 8, 9 and 10 are 100%, which indicate all users were able to complete the tasks, nevertheless task 4 and 5 are 80%, which caused by a user that was not able to finish the task, P4. According to Table 4, it can be said that the user completion level about tasks given was 96%, which means user effectiveness level during application process of e-finance was above average, since the normal average of completion task is 78% [22].

2) Error Rate
Error rate or the number of deffect, statistically relates to the number of user existing opportunities. The number of errors for each task conducted by the user during completion process of the task. Table 4 presents error rate of each task.
Based on Table 4, it can be understood the average of error rate when the user tries to complete the task is no more than 0.7%, thus error rate perfomed by 5 users upon 10 tasks can be concluded as adequate.

2) Efficiency
This metric measures the time spent in user need to complete the task. Table 5 shows the user's time in completing each task given, the time unit used is in seconds. A red line indicates that the assignment can not be properly disassembled by the user or the user concedes up before completing the task. Based on Table 5 it is found that the overall user relative efficiency calculation of the e-finance application is 92%.

Questionnaire
Questionnaire method was used to complement user testing method which is measuring usability level of user satisfaction aspect. The average SUS score of 5 users is 70. The average score of SUS with the value of 70 belongs to the category "OK". To achieve the "GOOD" or "EXCELLENT" or "BEST IMAGINABLE" level [23], it is necessary to make improvements to e-finance. This is significant, since the application is integrated with all village in the government of Malang City used for financial transactions.

Classification Problem
This part illustrates the issues or problems found on usability testing results (usability testing) that would be grouped based on similarity issues. Table 6 presents several problems found during usability evaluation of e-finance.

MA_9
There is a tabulation on the BAST filling form, even when the form filling is not used. Efficiency (task time)

MA_10
There is a tabulation on the SPP charging form, even when the form is not used. Efficiency (task time) Efficiency (task time)

MA_11
There is a tabulation on the SPM form, even when the form is not used Efficiency (task time)

MA_12
There is a tabulation on the SPP verification form, even when the form is not used Efficiency (task time)

Conclusion
This study developed a framework to evaluate the usability of e-government, which consist of 8 steps: (1) determining evaluation objectives, (2) determining usability aspects, (3) determining usability metrics , (4) selecting candidate usability evaluation method, (5) determining the required criteria of the method to be evaluated, (6) evaluating the method, (7) selecting and making the instrument, and (8) evaluate usability.
The result of the framework implementation of e-finance case study had found two combined methods, namely user testing and questionnaire. User testing was engaged to measure the level and identify problems in the aspect of effectiveness and efficiency, while the questionnaire was used to measure the usability level on the aspect of user satisfaction. The result of the framework implementation shows that the effectiveness, efficiency, and user satisfaction are 96%, 92%, and 70 respectively.
Fo further research, The proposed framework may be evaluated to further assess the proposed framework.. Beside, the framework evaluation also can be conducted by implementation toward the other case study by the different type of e-government.